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20 TECHNICAL FIELD 

The invention relates to a method for converting existing film or 
videotape motion pictures to a form that can be used with three- 
dimensional systems for broadcast or exhibition. 

25 



B ACKGROUND ART 

1 

30 With the advent of stereophonic sound, various techniques were 
developed to convert or 're-process 1 existing monophonic programs 
for stereophonic broadcast or recording systems. These included 
modifying the equilization, phase or tonal qualities of seperate 
copies of the monophonic program for the left and right channels, 

35 While true stereophonic or binaural effects may not have been 
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1 achieved, the effects were much improved over feeding the 
identical monophonic signal to both channels. 

Similarly, with the almost universal use of color production, 
5 exhibition and broadcast systems for motion picures and 
television, systems have been developed to convert existing 
monochrome or black 4 white materials to color programs. Such a 
system is described in applicant Geshwind's patent number 
4,606,625, issued August 19, 1986. The results of these colorized 
10 products, while not always identical to true color motion 
pictures, are more suitable than black & white for color systems. 



There have been a number of systems for exhibition or display of 
left- and right-eye pairs of binocular motion pictures. Early 
15 systems reqired two completely redundant projection or display 
systems; e.g. two film projectors or CRT television displays, 
each routed to one eye via mirrors. Other systems require either 
complicated and expensive projection or display sy terns, or 
expensive 'glasses 1 to deliver two seperate images. For example: 

20 

red- and green-tinted monochrome images are both projected 
or, displayed to be viewed through glasses with left and 
right lenses tinted either red or green; 

25 two full-color images are projected through mutually 

perpendicular polorized filters and viewed through glasses 
with lenses that are also polarized in the same manner; 

left and right images are displayed on alternate odd and 
30 even fields (or frames) of a standard (or high scan rate) 

television CRT and are viewed through 'glasses' with 
shutters (either rotating blades or flickering LCDs, for 
example) that alternate the view of left and right eyes in 
synchrony with the odd or even fields of the CRT. 
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Of the above systems, the second is not at all usable with 
standard home television receivers, the third requires very 
expensive 'glasses' and may flicker with standard home receivers, 
and the first produces only strangly tinted monochrome images. 
Further, none of the systems may be broadcast over standard 
television for unimpeded viewing without special glasses. 



Thus, until now, compatable (i.e., viewable as two-dimensional, 
without glasses) home reception of 3-D images was not possible. 

10 However, a new system, which takes advantage of differential 
processing of left- and right-eye images in the human perceptual 
system, delivers a composite image on a standard home television 
reciever that can be viewed as a normal 2-D picture without 
glasses. Very inexpensive glasses, with one light and one dark 

15 lens, accentuate the differential processing of the image, as 
viewed by each eye, to produce a 3-D depth effect. 

Practical, inexpensive, compatable (with standard TV) 3-D 
television may now become widespread. In addition to materials 
20 specifically produced for the new system (or other 3-D systems) 
conversion of standard 2-D programs to 3-D format would provide 
additional product to broadcast using the new compatable system 
(or for other 3-D projection systems). 

25 



SUMMARY OF THE INVENTION 
* 

Applicant Handal's previous application, number 479,679, filed 
30 March 28, 1983 and now abandoned, relates a process and apparatus 
for deriving left- and right-eye pairs of binocular images, from 
certain types of two dimensional film materials. In particular, 
the materials must consist of seperate foreground and background 
elements, such as cartoon animation cells and background art. By 
35 taking into account the parallax between scenes as viewed by the 
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1 left and right eyes, two images are prepared where foreground 
elements are shifted with respect to background elements by an 
amount that indicates their depth in the third dimension. Two- 
dimensional motion pictures that consist of a series of sinqle 

5 composite images could not be converted to three-dimensional 
format, by this technique, without first being seperated into 
various background and foreground elements. 



Once committed to two-dimensional motion pictures, the seperation 
10 and depth information for various scene elements, in the third 
dimension, are lost. Thus, the seperation of two-dimensional 
image sequences into individual image elements and the generation 
of three-dimensional depth information for each such image 
element are not simple or trivial tasks, and are the further 
15 subject of the instant invention. 

In accordance with the invention, standard two-dimensional motion 
picture film or videotape may be converted or processed, for use 
with three-dimensional exhibition or transmission systems, so as 

20 to exhibit at least some three-dimensional or depth character- 
istics. Seperation of a single 2-D image stream into diverse 
elements is accomplished by a computer assisted, human operated 
system. (If working from discrete 2-D film sub-componan t s, such 
as animation elements, the seperation step may be ommited.) Depth 

25 information is assigned to various elements by a combination of 
human decisions and/or computer analyses and resulting images of 
three-dimensional format are produced under computer control. 

* 

30 

BRIEF DESCRIPTION OF DRAWINGS 



A method for carrying out the invention is described in the 
accompanying drawings in which: 
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Figure 1 is a diagram illustrating the relationship of a 2-D 
source film image, left and right image pairs, and a 
composite 3-D image frame. 



5 



Figure 2 is a schematic diagram of a system for carrying out 
an impl imentation of the present invention. 
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DETAILED DESCRIPTION 



An immense amount of standard, 2-D motion picure material exists 
in the form of film and videotape. In addition, certain materials 
exist in the form of discrete 2-D image sub-componants, such as 
15 animated cell and background paintings. As the use of 3-D 
exhibition and broadcast systems becomes more widespread, the 
conversion of existing 2-D programs to a format that will exhibit 
at least some 3-D or depth effects, when used with 3-D systems, 
is desired. 



Extracting individual image elements or 3-D depth information 
from a. ( 2-D film frame, or synthesizing 3-D information for those 
elements, entirely by computer equiped with artificial 
intelligence, is not now practical. Therefore, the embodiment of 
25 the invention as described herein employs a high degree of human 
interaction with the computer. However, as artificial 
intelligence progresses, a predominantly or completely automated 
system may become practical and is within the intended scope of 
the invention. 



Figure 1 shows a frame 20 of a standard 2-D film or video motion 
picture, consisting of a cross-hatched background plane 21, a 
large white circle 22 in the mid-ground, and a small black square 
23 in the foreground. It is a 2-D representation of original 3-D 
35 scene 10 comprising elements 11, 12 and 13, which is not 
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1 available for direct 3-D photography. After human identification 
of individual image elements and depth assignment, the computer 
generates, from frame 10, a pair of coordinated left 30 and right 
40 images with backgrounds 31 and 41, circles 32 and 42, and 

5 squares 33 and 43 respectively. Note that the relative positions 
of the square and circle are different in the left and right 
images; this situation is similar to the parallax that might 
result between left- and right-eye views if one were to have 
viewed the original scene 10 directly. Three-dimensional format 
10 frame 50 is generated by encoding the information in the left 30 
and right 40 image pair, in a manner consistent with any one of a 
number of existing 3-D systems. The specific operation of these 
various 3-D systems is not the subject of the instant invention. 

15 Alternately, the steps of generating and encoding 3-D information 
may be combined such that"3-D format frame 50 may be processed 
directly from 2-D frame 20 without generating left 30 and right 
40 image pairs. In either case, 3-D format frame 50 when viewed 
by human 60 through 3-D glasses 70 is perceived as 3-D scene 80, 

20 containing elements 81, 82 and 83, which has at least some of the 
3-D characteristics of original 3-D scene 10. 

Various systems for the encoding, display, projection, recording, 
transmission or viewing of 3-D images exist, and new sysems may 

25 be developed. Specifically, various techniques for specifying, 
encoding and viewing 3-D information may now, or come to, exist, 
which do not make use of parallax offset and/or left and right 
image pairs and/or viewing glasses, or which embody new 
techniques or changes and improvments to current systems. 

30 Further, such systems may integrate information from more than 
one 2-D source frame 20 into a single resultant 3-D frame 50. The 
specifics of operation of such systems is not the subject of the 
instant invention, however, preparation of 2-D program material 
for such systems is. 
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1 The offsets shown for elements 31, 32 and 33 in left frame 30 and 
elements 41, 42 and 43 in right frame 40 are meant to be 
illustrative and do not necessarily follow the correct rules for 
image parallax. In fact, depending upon where viewer attention is 

5 meant to be centered, different rules may apply. For example, one 
technique is to give no parallax offset to far background 
elements and to give progressively more parallax offset to 
objects as they get closer. Alternately, attention may be 
centered in the mid-ground with no parallax offset to mid-range 
10 objects, some parallax offset to close-range objects and reverse 
parallax offset to far-range objects. The particular placement of 
objects and attention point in the 3-D scene is as much an art as 
a science and is critical to the enjoyment of 3-D programs and, 
in any event, is not meant to be the subject of this invention. 

15 

Figure 2 shows a schematic of a system to implement the instant 
invention. A 2-D film or video image 10 is input to a video 
monitor 20 and to the scanning portion 41 of frame buffer 40. 
Video monitor 20 is capable of displaying either the 2-D image 
20 being input 10 or the output from display portion 42 of frame 
buffer 40. 

•» 

Frame buffer 40 consists of an image scanner 41 which can convert 
the input image into digital form to be stored in a portion of 
25 frame buffer memory section 44, a display section 42 which 
creates a video image from the contents of a portion of memory 
section 44, and^a computer interface section 43 which allows the 
computer CPU 50 to read fropi and write to the memory section 44. 

30 Graphic input tablet 30 and stylus 35 allow the human operator to 
input position information to the computer 50 which can indicate 
the outline of individual image elements, choice of a specific 
image element or part of an image element, depth specification, 
choice of one of a number of functions offered on a 'menu 1 , or 

35 other information. An image cursor can be displayed on monitor 20 
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1 by frame buffer 40 to visually indicate the location or status of 
the input from the tablet 30 and pen 35. Text and numeric 
information can also be input by the operator on keyboard 55. 

5 Computer CPU 50 is equiped with software which allows it to 
interpret the commands and input from the human operator, and to 
process the digitized 2-D information input from 2-D frame 10 
into digital 3-D frame information, based on said human commands 
and input. Said digital 3-D frame information is then output by 
10 output interface 60 (which may be similar to frame buffer 40 or 
of some other design) to a videotape recorder 70 or to a film 
recorder 75, capable of recording 3-D format frames. 

The system as described above operates as follows. A frame of the 

15 2-D program is displayed on the video monitor for viewing by the 
human operator. With the tablet and stylus the operator outlines 
various 2-D image areas to indicate to the computer the 
boundaries of various image elements to be seperately processed. 
(For materials, such as animation componants, that already exist 

20 as seperate elements, the previous stage of the process may be 
skipped.) Depth position information, in the third dimension, is 
determined for each image element, by a combination of operator 
input and computer analysis. Left and right image pairs or a 3-D 
composite image is processed by the computer, from the 2-D input 

25 image, based on the computer software and operator instructions. 
Depending upon the particular 3-D system to be used, left- and 
right-image pairs may or may not be the final stage or an 
imtermediate stage or bypagsed entirely. Further, for some 3-D 
systems, information from more than one 2-D source frame may be 

30 combined into one 3-D frame (or frame pair). In any event, the 
final 3-D information is then collected on a videotape or film 
recorder. The process is then repeated for additional 2-D frames. 

Each image element may be given a uniform depth designation which 
35 may cause the perception of 'cardboard cut-out 1 characters. 
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1 Alternately, different portions of a single image element may be 
given different depth designations with the computer interpo- 
lating depth coordinates over the entire element. For example, an 
image element positioned diagonally in a frame may have its right 
5 edge designated to be closer than its left edge. Alternately, one 
feature of an image element, say a person's nose in a close-up, 
might be designated as being closer than the rest of the image 
element. In such manner, the computer would be instructed to 
interpolate and process depth information over the entire image 
10 element. Such processing of the image element, in accordance with 
varying depth information, may result in the stretching, skewing 
or other distortion of the image element. 

Depth interpolation may also be carried out over time, between 
15 frames. For example, an image element might be designated to be 
close in one frame and to be far away in a later frame. The depth 
of the image element may be interpolated for the frames between 
the two designated frames. Further, the position of the image 
element in the two dimensions of the film frame, and the shape of 
20 the outline seperating the image element from the rest of the 
source image, may also be interpolated over time, between frames. 
Linear , and non-linear interpolation techniques are well known and 
may be readily applied. 

25 It is also possible to add random noise to the depth information 
to eliminate the appearance of flat objects moving in space and 
to help achieve greater realism. 

. 

For a particular image element, depth position may be derived by 
30 the computer, alone or with operator input, by measuring the size 
of the image element in a frame. For example, once a person were 
outlined, his overall height in the 2-D film frame might be 
extracted by the computer as an indication of depth distance. 
Using knowledge, or making assumptions, about the object's real 
35 size, and the characteristics of the camera and lens, the 
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1 distance of the object from the camera may be calculated by 
applying well known principles of perspective and optics. 
Alternately, if overall size cannot be easily extracted by the 
computer, key points might be indicated by the operator for the 

5 computer to measure. Comparing the change of size of an image 
element in several frames will provide information about the 
movement of the object in the third dimension. 

It should be noted that horizontal parallax offsets are, by far, 
10 more obvious, due to the fact that our eyes are seperated in the 
horizontal direction. However, vertical offset differences 
between left- and right-eye views may also be appropriate. 

As various image elements are seperated and assigned depth 

15 values, a situation develops where diverse objects exist in a 
'three-dimensional space' within the computer. It should be noted 
that, in order to display a realistic representation of the 
entire scene, fowardmos»t objects must obscure all or part of 
rearmost objects with which they overlap (except in the case 

20 where a fowardmost object were transparent). When generating 
left- and right-eye views, the pattern of overlap of image 
elements, and thus the pattern of obscuring of image elements 
will, in general, be different. Further, for image elements that 
have been assigned non-uniform depth values (e.g., image elements 

25 that are not flat or not in a plane perpendicular to the third 
dimension) the intersection between these image elements, for the 
purpose of one obscuring the other, may be non-trivial to 
calculate. However, there are well known techniques, that have 
been developed for computer image synthesis, that allow for the 

30 sorting, intersection and display of diverse, overlapping 3-D 
image elements. 

As image elements are seperated from background scenes or each 
other, holes may develop. The computer software will compensate 
35 for this by filling in missing information in one particular 2-D 
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1 frame with the equivalent part of the image element from earlier 
or later frames whenever possible. Alternately, material to 'plug 
holes 1 may be created by the operator or by the computer from 
adjacent areas or may be newly synthesized. Further, as two image 

5 elements are seperated from each other, the fowardmost of the two 
may be moved closer to the viewer; hence, it may be appropriate 
to make it larger. Thus, the increased size may cover the gaps in 
the rearmost element of the two. 

10 As part of the processing to be performed on the 2-D source 
image, individual image elements or composite images, additional 
effects may be programmed into the computer to heighten the sense 
of depth. For example, shadows cast by one image element on 
another element or the background may be calculated and inserted; 

15 far distant landscape elements may be made hazy or bluish to 
indicate remoteness; different image elements may be blurred or 
sharpened to simulate d e p th- o f - f i e 1 d ; or, the distance between 
close objects may be ex^agerated to emphasize their seperation. 
There are, of course, other technical or artistic techniques that 

20 can be used to indicate depth in an image and which may also be 
incorporated into the image processing programs and would 
therefore be part of the invention as described herein. 
Therefore, the above examples are illustrative and should not be 
construed as limiting the scope of the invention. Alternately, 

25 depth information may be intentionally distorted for effect or 
for artistic purposes. 

Improvments may be made tg the final product by including new 
image elements that were not part of the original 2-D source 

30 image. These could include 2-D image elements that are then 
assigned depth values, 3-D image elements created by 3-D 
photography and then entered into the computer as left- and 
right-image pairs, for example, or synthetic 3-D computer 
generated graphics. In particular, since computer generated image 

35 elements can be created with depth information, they can be 




-12- 

1 easily integrated into the overall 3-D scene with vivid effect. 
For example, a 3-D laser blast could be created by computer image 
synthesis such that it would in turn obsure and be obscured by 
other image elements in an appropriate maner, and might even be 

5 created so as to appear to continue beyond the front of the 
screen into 'viewer space 1 . 

Animated film componants usually have relatively few background 
paintings, which are kept seperate from the animated characters 

10 in the foreground. For these, or for live 2-D filmed scenes, once 
the foreground elements have been seperated from the background, 
the flat 2-D backgrounds may be replaced by 3-D backgrounds. The 
3-D backgrounds might consist of computer generated graphics, in 
which case depth information for the various elements of the 

15 background would be available at the time of the background 
creation. Alternately, 3-D backgrounds might be created using 3-D 
photography, in which case depth information for the background 
elements may be derived,* by the computer, from the comparison of 
the left- and right-image pairs of the 3-D background photographs 

20 and applying known image processing and pattern recognition 
techniques. Alternately, depth information to create 3-D 
backgrounds may be specified otherwise by operator input and/or 
computer processing. Once depth information is available for the 
various background elements, it may compared to depth information 

25 for the other image elements with -which it is to be combined in 
each frame. Thus, the 3-D background may itself have some depth 
and, in effect, be a 'set 1 within which other image elements may 
be positioned in front o/, behind or intersecting various 
background elements for added realism. 

30 

It should be noted that, intended purpose and software algorithms 
aside, the design and operation of the 3-D conversion system 
described herein has many similarities with the black & white 
footage colorization system described in applicant Geshwind's 
35 patent A, 606, 625. They are both computer-aided systems which 
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1 display standard film images to an operator, allow the operator 
to input information seperating various image elements within the 
frame, allow the operator to specify attributes (color in one 
case, depth in the other) for the image elements, and cause the 

5 computer to process new image frames from the original, based on 
the operator input. Thus, the processing of 2-D black and white 
footage to add both color and depth information would be much 
more efficient than implementing each process seperately. 

10 The scope of the instant invention is the conversion or 
processing of 2-D program material for use with any 3-D 
exhibition or distribution system now in use or later developed, 
but not the specific method of operation of any particular 3-D 
system . 

15 

It will thus be seen that the objects set forth above, among 

those made apparent from the proceeding description, are 

efficiently attained anc} certain changes may be made in carrying 

out the above method and in the construction set forth. 

20 Accordingly, it is intended that all matter contained in the 

above description or shown in the accompanying figures shall be 

interpreted as illustrative and not in a limiting sense, 
-i 

Now that the invention has been described, what is claimed as new 
25 and desired to be secured by Letters Patent is: 
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